Skip to content

feat(data): record which feed a cached series came from (#696, part 1… - #710

Merged
eaitbrahim merged 2 commits into
mainfrom
feat-696-partial-feed-refuses
Sep 3, 2026
Merged

feat(data): record which feed a cached series came from (#696, part 1…#710
eaitbrahim merged 2 commits into
mainfrom
feat-696-partial-feed-refuses

Conversation

@eaitbrahim

Copy link
Copy Markdown
Contributor

… of 2)

Schema v17. The provenance half of #696, landing BEFORE the gate that reads it -- a gate built on a feed inferred at read time would encode the bug it exists to fix.

THE BUG. keel's liquidity statistic is median(volume * close) over cached candles, and every threshold keyed on it was calibrated against venue-reported volume. That is coherent on a crypto exchange: Coinbase's own volume IS the scale the floor was chosen against. It is not coherent on Alpaca's IEX feed, which reports one US equity exchange's own executions. The cost-fidelity run measured MSFT -- one of the most liquid securities in the world -- cached at $186M/day against a model anchored at $500M, and priced it as a thin asset. Nothing recorded which feed produced those bars, so the answer was whatever broker.data_feed happened to be loaded when someone later read the series.

WHAT LANDS HERE.

candle_series_feed, keyed on (product_id, granularity, feed) -- NOT a mutable column on candles. A series fetched under both feeds records BOTH rows, because that is what happened; a single overwritable column would let the most recent fetch erase the fact that most of the bars came from somewhere narrower, and a mixed series is exactly the case a reader most needs warning about.

NO BACKFILL, and that is the point rather than a shortcut. Seeding rows from the current broker.data_feed would manufacture precisely the claim the table exists to make checkable, for years of bars that may have come from elsewhere. An empty table means "unrecorded", which is TRUE of every pre-existing series and is a different statement from "consolidated" -- reports_consolidated_volume returns None for the first and False/True for the second, and callers must not conflate the absence of evidence with evidence.

keel/data/feed_scope.py carries the design, whose core is an ASYMMETRY: a single-venue feed's volume is a LOWER BOUND on consolidated volume, never an upper one. Volume at or above the floor on a partial feed is therefore a CONCLUSIVE pass -- if a name traded that much on one venue alone it necessarily traded at least that much in total -- while volume below the floor licenses no claim either way and must be refused as unmeasured rather than reported as thin. That is what lets this be honest without encoding any venue's market share: the bound holds for any share below 100%, so nothing here has a percentage in it that market structure could invalidate. test_the_bound_needs_no_market_share pins that no figure leaks into executable code, because the one thing that must never happen is someone scaling a volume statistic by an assumed share.

An unrecognised feed id reads as PARTIAL, which costs a conclusive pass on a consolidated feed nobody declared and never grants one on a narrow feed nobody declared.

Adapters declare their own feed (volume_feed_id), qualified by venue -- alpaca:iex / alpaca:sip, coinbase -- never bare iex, which would collide the day another venue routes there. Declaration is structural and optional: an adapter that declares nothing records nothing, so unrecorded stays unrecorded rather than every crypto series silently acquiring a provenance nobody established. Forcing all five adapters to implement it would invite a placeholder, which is the one value that must never enter this table.

Tests: 31 across three files.
tests/data/test_series_feed_provenance.py -- 16, the storage contract
tests/data/test_feed_scope.py -- 11, the scope verdict
tests/data/test_feed_provenance_is_wired.py -- 4, the fetch path actually
carries the declaration; without these the whole thing degrades silently to
pre-issue behaviour with a schema and a module suggesting otherwise

The provenance and wiring suites were written test-first. feed_scope.py was NOT -- I wrote it before its tests, which is the wrong order; the 8-mutant run below is what establishes those tests constrain the code rather than merely describe it.

Mutation-verified, 8 mutants, all killed: unrecorded reading as consolidated; any for all on a mixed series; alpaca:iex declared consolidated; a blank declaration accepted; provenance recorded for an empty batch; an empty feed string accepted; a re-fetch overwriting first_seen_ts; a missing table raising instead of reading empty.

Five existing version pins move 16 -> 17. They are literals on purpose -- a deliberate speed bump that makes a schema bump a decision someone acknowledged.

Refs #696

Claude-Session: https://claude.ai/code/session_01NzuKAe2RVrPt9acVAWjRyL

What & why

Tests-first evidence

  • Tests written first, seen failing for the right reason

Gates (all must pass)

  • uv run ruff check clean
  • uv run mypy clean
  • uv run pytest -q green

Scope check

  • This PR touches a rail or a default classification — checked means it DOES;
    leave checked only if true, and if so: cite the source and open the discussion
    BEFORE review (CONTRIBUTING.md, "Governance: rulings vs. machinery").
  • New dependency added (needs discussion first)

eaitbrahim and others added 2 commits September 3, 2026 12:32
… of 2)

Schema v17. The provenance half of #696, landing BEFORE the gate that reads it --
a gate built on a feed inferred at read time would encode the bug it exists to
fix.

THE BUG. keel's liquidity statistic is `median(volume * close)` over cached
candles, and every threshold keyed on it was calibrated against venue-reported
volume. That is coherent on a crypto exchange: Coinbase's own volume IS the scale
the floor was chosen against. It is not coherent on Alpaca's IEX feed, which
reports one US equity exchange's own executions. The cost-fidelity run measured
MSFT -- one of the most liquid securities in the world -- cached at $186M/day
against a model anchored at $500M, and priced it as a thin asset. Nothing
recorded which feed produced those bars, so the answer was whatever
`broker.data_feed` happened to be loaded when someone later read the series.

WHAT LANDS HERE.

`candle_series_feed`, keyed on `(product_id, granularity, feed)` -- NOT a mutable
column on `candles`. A series fetched under both feeds records BOTH rows, because
that is what happened; a single overwritable column would let the most recent
fetch erase the fact that most of the bars came from somewhere narrower, and a
mixed series is exactly the case a reader most needs warning about.

NO BACKFILL, and that is the point rather than a shortcut. Seeding rows from the
current `broker.data_feed` would manufacture precisely the claim the table exists
to make checkable, for years of bars that may have come from elsewhere. An empty
table means "unrecorded", which is TRUE of every pre-existing series and is a
different statement from "consolidated" -- `reports_consolidated_volume` returns
`None` for the first and `False`/`True` for the second, and callers must not
conflate the absence of evidence with evidence.

`keel/data/feed_scope.py` carries the design, whose core is an ASYMMETRY: a
single-venue feed's volume is a LOWER BOUND on consolidated volume, never an
upper one. Volume at or above the floor on a partial feed is therefore a
CONCLUSIVE pass -- if a name traded that much on one venue alone it necessarily
traded at least that much in total -- while volume below the floor licenses no
claim either way and must be refused as unmeasured rather than reported as thin.
That is what lets this be honest without encoding any venue's market share: the
bound holds for any share below 100%, so nothing here has a percentage in it that
market structure could invalidate. `test_the_bound_needs_no_market_share` pins
that no figure leaks into executable code, because the one thing that must never
happen is someone scaling a volume statistic by an assumed share.

An unrecognised feed id reads as PARTIAL, which costs a conclusive pass on a
consolidated feed nobody declared and never grants one on a narrow feed nobody
declared.

Adapters declare their own feed (`volume_feed_id`), qualified by venue --
`alpaca:iex` / `alpaca:sip`, `coinbase` -- never bare `iex`, which would collide
the day another venue routes there. Declaration is structural and optional: an
adapter that declares nothing records nothing, so unrecorded stays unrecorded
rather than every crypto series silently acquiring a provenance nobody
established. Forcing all five adapters to implement it would invite a
placeholder, which is the one value that must never enter this table.

Tests: 31 across three files.
  tests/data/test_series_feed_provenance.py   -- 16, the storage contract
  tests/data/test_feed_scope.py               -- 11, the scope verdict
  tests/data/test_feed_provenance_is_wired.py -- 4, the fetch path actually
      carries the declaration; without these the whole thing degrades silently to
      pre-issue behaviour with a schema and a module suggesting otherwise

The provenance and wiring suites were written test-first. `feed_scope.py` was
NOT -- I wrote it before its tests, which is the wrong order; the 8-mutant run
below is what establishes those tests constrain the code rather than merely
describe it.

Mutation-verified, 8 mutants, all killed: unrecorded reading as consolidated;
`any` for `all` on a mixed series; `alpaca:iex` declared consolidated; a blank
declaration accepted; provenance recorded for an empty batch; an empty feed
string accepted; a re-fetch overwriting `first_seen_ts`; a missing table raising
instead of reading empty.

Five existing version pins move 16 -> 17. They are literals on purpose -- a
deliberate speed bump that makes a schema bump a decision someone acknowledged.

Refs #696

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NzuKAe2RVrPt9acVAWjRyL
…ch` (#696)

Four findings from the review of #710, fixed on the branch. The first one meant
the feature was very nearly inert.

1. THE ONE THAT MATTERED. Only `history.ensure_history` carried the feed
   declaration. `market_feed.poll_once` -- the path `agent.run_once` uses on EVERY
   cycle in every deployment -- and `repair.repair_series` both dropped it, so in
   a real database almost every bar would have been written with no provenance
   and `candle_series_feed` would have stayed nearly empty while a schema, a
   module and 31 tests said otherwise. `keel fetch` is not how bars normally
   arrive.

   The wiring tests did not catch it because they exercised `ensure_history`
   alone: they proved the mechanism worked, never that every writer used it. Both
   paths now resolve the feed once and thread it down, and there are tests per
   WRITER rather than per mechanism. `backfill` resolves its own rather than
   relying on one being threaded in -- it has no in-tree caller today, and a
   writer that silently records nothing is precisely the failure this table
   exists to prevent.

2. `get_series_feeds` / `get_series_feed_window` rescued `sqlite3.OperationalError`
   wholesale, which is also what a LOCK TIMEOUT raises -- and keel reads and
   writes this file from more than one process by design. A lock would have been
   reported as "scope unrecorded" for a series whose scope is on disk: exactly the
   `None`-vs-`False` conflation `feed_scope` exists to prevent, silently. Now only
   `no such table` is rescued; anything else propagates.

3. `DeclaresVolumeFeed` was declared, exported and never used, because
   `volume_feed_of` reads the attribute with `getattr` (presence is not enough --
   the value must also be a non-blank string). A Protocol nothing checks drifts
   from the check that matters, so a test now asserts both adapters satisfy it.

4. Three closing parens sat at column 0 from a scripted edit. `ruff check` passes
   on them because `ruff format` is not a CI gate, which is why they survived.

Mutation-verified, 4 mutants. One SURVIVED on the first pass: reverting the
narrowed rescue in (2) to a bare `return ()` broke nothing, because I had fixed
the catch without writing a test that a lock must propagate.
`test_a_locked_database_is_raised_not_read_as_unrecorded` closes that, and all
four now die.

Refs #696

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NzuKAe2RVrPt9acVAWjRyL
@eaitbrahim
eaitbrahim merged commit 78123ff into main Sep 3, 2026
4 checks passed
@eaitbrahim
eaitbrahim deleted the feat-696-partial-feed-refuses branch September 3, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant